Proposal PR: CI-based performance testing with Docker runner #43

GroophyLifefor · 2025-08-13T15:59:03Z

Performance Runner Proposal

Docker-based performance testing framework for JavaScript packages with CI integration and multi-Node.js version support.

Summary

Containerized test runner that provides:

Consistent testing environments across Node.js versions (18-24)
Automated CI/CD workflows with PR commenting
Template system for common performance testing patterns
Standardized result formatting

Quick Setup

Add CI workflow using the Docker image
Create tests in expf-tests/ directory
Use provided templates (autocannon example included)

For detailed one, read /perf-runner/readme.md.

Full Details

See the complete problem statement and technical specifications in Issue #38.

wesleytodd

Awesome to see this work. I think a lot of this would be great as incremental PRs into the existing CLI setup we have merged. Are you up for that? Are the windows issues from #45 blocking you? Would we be able to do the smaller incremental PRs for that to move that forward? Or is it something else blocking integrating this work into the existing CLI?

wesleytodd · 2025-08-20T16:07:37Z

perf-runner/Dockerfile

@@ -0,0 +1,24 @@
+FROM debian:bullseye-slim


I would take a look at what the existing docker runner is doing. It uses the official node.js docker images. I think we want to stick to using supported images from either node.js, GH, or our partners like NodeSource.

https://github.com/expressjs/perf-wg/blob/main/packages/runner-docker/Dockerfile

Good point, also this can be a optimization that I'm not sure but valid point.

I don't think we should optimize system things, we should use what our users would use with the goal of being "close to the platform". That way we may catch perf issues from common configurations before users do (or at least can easily test them).

wesleytodd · 2025-08-20T16:09:05Z

perf-runner/run-tests.mjs

@@ -0,0 +1,62 @@
+import { config, validateConfig } from './src/config.mjs';


I think this is a good start for a runner-local package. Maybe take a look at what I did here and follow that pattern? https://github.com/expressjs/perf-wg/blob/main/packages/runner-docker/index.mjs

Answered in PR Comment: #43 (comment)

wesleytodd · 2025-08-20T16:10:38Z

perf-runner/src/github-api.mjs

+/**
+ * Post a comment to a GitHub PR
+ */
+export async function postComment(message) {


If we are using GHA we can just rely on their PR checks instead of making comments right? They even have an html reports: https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-commands#adding-a-job-summary

I didn't know that, I'll look into it, thank you.

wesleytodd · 2025-08-20T16:11:49Z

perf-runner/src/result-processor.mjs

@@ -0,0 +1,137 @@
+import { readFileSync } from 'fs';


It would be awesome to see this as an incremental addition to the existing compare command. Especially awesome to see it output the node version and other things you have in here.

Answered in PR Comment: #43 (comment)

GroophyLifefor · 2025-08-20T16:47:22Z

Awesome to see this work. I think a lot of this would be great as incremental PRs into the existing CLI setup we have merged. Are you up for that? Are the windows issues from #45 blocking you? Would we be able to do the smaller incremental PRs for that to move that forward? Or is it something else blocking integrating this work into the existing CLI?

First of all, you are reading my mind.

This is how I see it: I think what you did was great, but it was a heavy introduction, and then I made a big mistake by complicating things further under the cover of Windows support.
Then I made CI based perf-runner because, yeah, if you look the PR pretty less code with more feature.
BUT just because I presented an approach, putting everything aside and moving on to the structure I had established caused contradictions between what was reasonable and what was right. And so I started thinking about how we could combine the two approaches.

Right now, I have no idea, sorry 😄 , but mine target is simply:

isolated, easier to maintain, splitable to repositories, versioning(as Dockerimage supports)

And your target is simply:

local-first, cli-based, everything on perf repo

The base conflict I see

mine: splitable to repositories
your: everything on perf repo

I'm thing to create a testing folder for repo spesific testings rather than perf-wg repository setup.

Whose can be combined

ci-based and cli-based: Currenly my approach is only-CI solution but since I may can't do cli-based, I may can do cli-supportive.
ci-based and local-first : As above, if we do cli-supportive, also can be local performance test support.

perf-runner/readme.md

Co-authored-by: Sebastian Beltran <[email protected]>

wesleytodd · 2025-08-22T13:19:35Z

local-first, cli-based, everything on perf repo

Yes, I did this for what I think are good reasons:

local-first: because we need to be able to iterate on things fast and it is much easier to take a cli/local setup and call that from CI than to design a CI system and add in local runs later.
cli-based: in addition to number 1, being cli based means we can much more easily give contributors instructions on how to run their own quick/one-off perf tests for PRs across the org. If we didn't have this, we would need folks to setup their forks to run CI and depending on the type of change would require forking other unnecessary repos (for example, if the change was in router but the tests target express).
everything on perf repo: I covered this a bit in Types of Performance Tests #12 & Standardized formats, locations, and data contracts #19, but for now having it all here helps us keep things organized. In the end, different types of tests would live in their project repos (see Types of Performance Tests #12 & Standardized formats, locations, and data contracts #19). Until we have all agreed on the formats and requirements though, I think multi-repo would really make it hard for collaboration.

Does number 3 help resolve this conflict for you? I was treating the tests in this repo as starting examples that we could move out once we are sure they are the right ones. That way, prevent having work happening all over the org's ~40 repos that we may have to update if we make core tooling changes.

Proposal PR: CI-based performance testing with Docker runner

0e83349

github-project-automation bot added this to Perf WG Tooling Aug 13, 2025

GroophyLifefor requested a review from a team August 13, 2025 15:59

GroophyLifefor self-assigned this Aug 13, 2025

GroophyLifefor added the enhancement New feature or request label Aug 13, 2025

GroophyLifefor linked an issue Aug 13, 2025 that may be closed by this pull request

Isolated and Consistent Performance Testing in CI #38

Open

fix: replace docker image author to placeholder

24599e6

wesleytodd requested changes Aug 20, 2025

View reviewed changes

bjohansebas reviewed Aug 20, 2025

View reviewed changes

perf-runner/readme.md Outdated Show resolved Hide resolved

fix(docs): Removed OpenJS mention in working group mention

fd0be46

Co-authored-by: Sebastian Beltran <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Proposal PR: CI-based performance testing with Docker runner #43

Proposal PR: CI-based performance testing with Docker runner #43

Uh oh!

GroophyLifefor commented Aug 13, 2025

Uh oh!

wesleytodd left a comment

Uh oh!

wesleytodd Aug 20, 2025

Uh oh!

GroophyLifefor Aug 20, 2025

Uh oh!

wesleytodd Aug 22, 2025

Uh oh!

wesleytodd Aug 20, 2025

Uh oh!

GroophyLifefor Aug 20, 2025

Uh oh!

wesleytodd Aug 20, 2025

Uh oh!

GroophyLifefor Aug 20, 2025

Uh oh!

wesleytodd Aug 20, 2025

Uh oh!

GroophyLifefor Aug 20, 2025

Uh oh!

GroophyLifefor commented Aug 20, 2025

Uh oh!

Uh oh!

wesleytodd commented Aug 22, 2025

Uh oh!

Uh oh!

		@@ -0,0 +1,62 @@
		import { config, validateConfig } from './src/config.mjs';

Uh oh!

Proposal PR: CI-based performance testing with Docker runner #43

Are you sure you want to change the base?

Proposal PR: CI-based performance testing with Docker runner #43

Uh oh!

Conversation

GroophyLifefor commented Aug 13, 2025

Performance Runner Proposal

Summary

Quick Setup

Full Details

Uh oh!

wesleytodd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GroophyLifefor commented Aug 20, 2025

Uh oh!

Uh oh!

wesleytodd commented Aug 22, 2025

Uh oh!

Uh oh!